Rafiki: A Middleware for Parameter Tuning of NoSQL Datastores for Dynamic Metagenomics Workloads

High performance computing (HPC) applications, such as metagenomics and other big data systems, need to store and analyze huge volumes of semi-structured data. Such applications o‰en rely on NoSQL-based datastores, and optimizing these databases is a challenging endeavor, with over 50 con€guration parameters in Cassandra alone. As the application executes, database workloads can change rapidly from read-heavy to write-heavy ones, and a system tuned with a read-optimized con€guration becomes suboptimal when the workload becomes write-heavy. In this paper, we present a method and a system for optimizing NoSQL con€gurations for Cassandra and ScyllaDB when running HPC and metagenomics workloads. First, we identify the signi€cance of con€guration parameters using ANOVA. Next, we apply neural networks using the most signi€cant parameters and their workload-dependent mapping to predict database throughput, as a surrogate model. Œen, we optimize the con€guration using genetic algorithms on the surrogate to maximize the workloaddependent performance. Using the proposed methodology in our system (Rafiki), we can predict the throughput for unseen workloads and con€guration values with an error of 7.5% for Cassandra and 6.9-7.8% for ScyllaDB. Searching the con€guration spaces using the trained surrogate models, we achieve performance improvements of 41% for Cassandra and 9% for ScyllaDB over the default con€guration with respect to a read-heavy workload, and also signi€cant improvement for mixed workloads. In terms of searching speed, Rafiki, using only 1/1000-th of the searching time of exhaustive search, reaches within 15% and 9.5% of the theoretically best achievable performances for Cassandra and ScyllaDB, respectively— supporting optimizations for highly dynamic workloads. ACM Reference format: Elided. 2016. Rafiki: A Middleware for Parameter Tuning of NoSQL Datastores for Dynamic Metagenomics Workloads. In Proceedings of ACM Conference, Washington, DC, USA, July 2017 (Conference’17), 13 pages. DOI: 10.1145/nnnnnnn.nnnnnnn